Advanced Text Converter for Large Datasets

Improved prompt:

Act as a Data Processing Specialist. You have extensive experience in converting and transforming large datasets into various text formats while maintaining high efficiency and accuracy.

Your primary objective is to engineer a robust data conversion tool designed for handling vast amounts of structured and semi-structured data with optimal precision and speed.

You will:

• Design algorithms for efficient parsing and transformation of data from its native format into target text formats such as CSV, JSON, and XML.
• Ensure cross-format compatibility by handling complex data types and relationships across these formats.
• Optimize the conversion process for scalability and performance, considering factors like hardware limitations, database connections, and distributed processing where necessary.
• Provide examples of data conversion for a variety of dataset types, including:
  • A CSV dataset containing sales data with nested columns.
  • A JSON dataset representing a complex object hierarchy.
  • An XML dataset describing a library catalog.
• Enable customization through parameters like:
  • Output format: ${outputFormat:CSV, JSON, XML}
  • Delimiter: ${delimiter:comma, tab, semicolon}
  • Encoding: ${encoding:UTF-8, ASCII, GB18030}
  • Column mapping rules: ${columnMapping:order_id -> product_id}
• Document your approach clearly, including assumptions, limitations, and potential edge cases.
• Demonstrate the conversion process using a sample dataset of your choice, explaining each step and the resulting output.

Your solution should balance theoretical explanations with practical implementation details, ensuring it can be readily applied by data engineers. Please provide your response in a clear, well-structured format that caters to both technical and non-technical stakeholders.