3 Challenges to Being a Data Scientist in 2024

Author:Murphy | View: 26995 | Time: 2025-03-22 21:03:18

It's a great time to be a data scientist right now. It is one of the fastest growing careers, with the projected increase in job openings from 2022 to 2032 at 35%. The average salary in the US is about $160,000 (depending on experience, of course). Huge strides are being made in the field, particularly in the Artificial Intelligence sector, with the advent of large language models (LLMs) providing a host of new tasks and opportunities.

But with all these perks come a set of unique challenges that data scientists are currently facing, and will continue to face in the coming years. Before you consider jumping into the field, it's important to be aware of some of these challenges so you can evaluate if this is the career path for you.

1. It's a rapidly developing field

Relative to other fields like finance and engineering, Data Science is still a developing field. Degrees in data science didn't start being offered widely until around 2012. Even in other more established fields, as modern technology and society evolves, there will always be change.

New tools, technologies, and languages are being developed every day. Python updates regularly and so do its standard packages like pandas and numpy.

Best practices shift and evolve, and new skillsets become increasingly desirable. One of the changes I've noticed recently is the push, and the demand for data scientists to become well versed in MLOps, learning platforms like Google Cloud and AWS.

And of course, with the recent advent of LLMs and AI, there is an increasing demand and push for data scientists to at least be familiar with and know how these tools work.

If you want to excel in this field, you need to dedicate a good amount of time to research, keeping up with new trends, and learning new skills. Having a degree is not enough, because job requirements can change in a matter of months.

2. Market conditions and competition

Tech layoffs have been an increasing topic of discussion. In 2023, almost 200,000 workers were laid off at US-based tech companies. So far in 2024 that number is approaching 60,000.

The good news is that data scientists have remained one of the safer positions, with only about 3% of the layoffs being data science jobs.

That being said, the risk is still very real and it's something you should be aware of, especially if you want to work for a large company like Google or Amazon.

Though job openings are increasing overall, so are eligible applicants. As tech giants who once attracted top talent lay off their employees, these highly experienced candidates enter the market, making it increasingly competitive.

The raw numbers of increasing job postings can be misleading as well. Just because there are x amount of openings each year for data scientists doesn't mean that data scientists (especially newer ones) aren't struggling in the current market.

To showcase this, I ran my own experiment using LinkedIn, searching for the key words "data scientist". I started off by not including any filters, which yielded over 7,000 job postings. But the reality is that most people are restricted by location, experience level, and position type (e.g. full time, internship, contract). So here are some of the ways that narrowing down my search, even reasonably, led to a steep decline in results.

Number of data science job postings on LinkedIn for various conditions. Image by author

Some notes so you can understand the chart labels I used:

The only filters I played with for this experiment were "Experience Level", "Job Type – eg Full time, Part Time, or Contract", "Location Type – eg Remote, Hybrid, or On Site", and "Location – eg. New York"
Any time a filter says "x only" it means this was the only filter I applied. So "full time only" included any location type, any experience level, and any location (all US) but filtered for full time positions only
Filters which have multiple conditions are listed, and these were the only filters applied. For example, "Remote, full time" only filtered for location type (remote) and job type (full time). All other filters such as experience were open to any.

Even by searching for full time positions only, you've cut your options in half from the original. If you search for remote positions, meaning you could work from anywhere in the US, you are down to about 700, though you're still better off trying to work remote from Colorado for a New York company than trying to find a company to hire you in New York only for any location type (remote, on site, or hybrid).

If you're entry level, this makes things even harder. Your options go from 758 to 74. Not to mention that many positions on LinkedIn labeled "entry level" really aren't entry level at all.

So yeah, it's rough out here. Long gone are the days when the words "data scientist" "Python" and "machine learning" on a resume are enough to grab a hiring manager's attention and stand out.

Nowadays you don't just need a resume. You need an online portfolio and website, a repository of unique projects, and potentially even an online presence to get noticed.

Even then, the competition is brutal. These data science job postings get hundreds, if not thousands of applicants. More and more people want to get into this field. So be prepared to work extra hard to stand out.

3. Constantly problem-solving

One of the things that makes data science a fun job is also in my opinion one of its downsides. Because it is a fast paced field full of constant change, not every problem you approach will be predictable or repeatable. Sure, the more years you put into this, the more intuition you'll stack up for certain types of problems.

But you will have many frustrating days, especially as a beginner. This isn't a job you can get trained for one and done. You have to learn how to think. It's similar to software engineering in that sense. There will always be a new kind of error you need to handle, a new bad kind of prediction your model makes, or a new way that data gets corrupted. There will be days where you spend the majority of your time trying to debug a script or retrain a model.

The challenge is fun at first but over time it can wear you down. You may be sitting down most of the day, but your mind will get tired. Sometimes you will long for a more mindless job that you can do with your eyes closed and muscle memory. But maybe that's just me.

If you're interested in learning more about what I personally do on a daily basis, I wrote an article explaining my typical work day in depth.

A Day in the Life of a Data Scientist

Conclusion

Data science is not for everyone. It is an increasingly competitive, rapidly changing, and highly mentally stimulating career. For some, this is their dream. Others may prefer a more stable job that they can master within 6 months and do repetitive tasks with more mental ease.

As AI evolves and becomes more integrated into the workplace, companies will increasingly need data scientists and machine learning engineers to understand these tools and manage them. People know this – it's why universities have ramped up their offerings of data science and AI courses and degrees. As the hype around AI continues to increase, more and more people are going to want to get into the field. You'll need to stay sharp, stay up to date, and continue to learn over and over if you want to be successful and make it in this field.

With all of these challenges come great rewards, as you will be part of building and maintaining technologies that are shaping the future. With data science and programming knowledge, you have immense power to change the world for good. Data is everywhere and is extremely valuable, which means you and your skillset are also going to be very valuable for a long time.

Just make sure that you know what you are getting into, think about the possible downsides, and evaluate if this career path is appropriate for your personal strengths and weaknesses.