(Editor’s note: This is the second article in a weekly series looking at the top jobs in data science and analytics. Part one looked at the demand and opportunities for data scientists. This article explores the companion role of the data engineer).
Job title: Data engineer
Reports to: As with a data scientist, the data engineer can again report into any number of various people within a business. The data engineer will often sit as part of a team with data scientists, building the infrastructure from which the data scientists will work, so whom they report into may be very similar. Typically, a data engineer will report into anyone from a lead data engineer, a director or vice president of engineering, a chief data engineer or architect or a chief technology officer.
Demand for this role: The data engineer has been voted third in a poll for “America’s top tech jobs” for the past two years, and demand for this type of candidate goes hand in hand with the data scientist. For every data scientist hired, there will be at least two data engineers building the vital infrastructure that is completely necessary in order for the data scientist to conduct their own work.
Following recent government policy changes in the H1-B Visa application process, demand for US citizen data engineers has increased drastically and shows no signs of easing. This newfound demand for data engineers could and most likely will drive higher salaries and perhaps increase the contract/consulting market in this space that is very much a permanent, full time dominant occupation.
Top industries hiring for this job: As with all top technology positions in the US right now, every industry is recognising the growing need for cutting edge big data platforms and infrastructure, and it is the data engineer that will provide these. Every business with large data lakes of information will require strong data engineers to build robust platforms to store, organize and ultimately utilize the data that they have collected, and companies that wish to build or enhance their data science functions will need to hire a number of data engineers for that to be successful, or even viable.
Responsibilities with this job: Unlike the data scientist, a data engineers’ role will be somewhat similar in most industries, meaning their skills are extremely transferrable between various sectors. The main responsibility of the data engineer is to build and maintain the infrastructure and data management systems that will store and structure all of a company’s data, as well as looking at improving the reliability and efficiency of these systems. They will often work closely with data architects and data scientists on a daily basis, and as mentioned they will provide the entire grounding from which businesses critical analyses can be conducted by the data scientists.
Required background for this job: Academic background is not regarded as important as the technical skills that a data engineer will have in their portfolio. A bachelor’s degree in anything from computer science, electrical engineering or similar will usually be all that an employer would look for, and it is the technologies and tools that a candidate is proficient with that is much more attractive. A typical data engineer will have Python or Java as their main programming language, and experience in big data technologies such as Hadoop, Spark, Hive, Pig and the like will be completely necessary. Exposure to cloud infrastructure, whether Amazon Web Services, Microsoft Azure or Google Cloud is also a huge bonus for employers as most businesses operate away from the desktop.
Skills requires for this job (technical, business and personal): A data engineer will need to be a candidate that enjoys remaining extremely technical and hands on, and will need to be collaborative by nature given the structure of the team around them. A must is passion for working with large data sets and building robust and efficient data management systems.
Compensation potential for this job: In the current New York City job market, a fresh graduate could expect a salary between $80,000 to $100,000, and this figure rises to between $150,000 to $180,000 for a data engineer with 5-8 years commercial experience.
Success in this role defined by: Given the nature of a data engineers’ responsibilities, success will be defined by building extremely efficient and easy to use data management systems and robust infrastructure that will improve the work of data scientists and all other users. The ability to always improve the performance and capability of the infrastructure will be a data engineer’s top priority and a good measure of your success.
Advancement opportunities for this job: Progression in this field can go in a number of directions. A typical career advancement would be to move into a lead engineer role and eventually into a VP or director of engineering, however you may choose to move into data architecture wherein you design the data management systems to be built by engineers, rather than build them yourselves. This may lead into a chief data architect position later on in your career. Many data engineers could also move into CTO level roles.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access