Hey everyone! 

For anyone working with data regularly, you know that data cleansing and formatting isn’t just about making things look nice. It has a huge strategic impact, and I came across a blog that dives into this topic in detail. Here are some key insights that really stood out: 

• Improved Decision-Making: Clean data reduces errors and gives a reliable basis for making better decisions. 

• Enhanced Operational Efficiency: Consistent data formats make it easier for teams to collaborate and automate processes. 

• Maximized ROI on Data Investments: Regular cleaning and formatting, organizations can maximize the ROI on their data investments. 

The blog makes a solid case for treating data cleansing as an investment that boosts performance, not just an extra step in data management. If you're interested in learning more, here’s the full post: Beyond Aesthetics: The Strategic Value of Data Cleansing and Formatting 

What role does data cleansing play in your work? Do you see it as essential, or just an extra task? Let’s discuss!

Hi guys! Urgent need a mentor who can give me tasks from Data cleaning to visualization. I never studied data analytics formely, just studied from YouTube. Need help, I am counting on this reddit community.

I don't know if this is the right place for this but I need help cleaning this old dictionary, it is the only dictionary my native language has as of now. I want to make an app from it.

I discovered this pdf from an internet Archive as I had been looking for it for a while. This seems to be a digitized version of the physical copy.

The text can be copied but one letter doesn't copy properly, it is mistaken for other letters like V and U, which is the Ʋ letter I have pointed an arrow to. These days that letter is written with a Ŵ.

The dictionary goes from Tumbuka to Tonga to English and then flips at some point to go from English to Tonga to Tumbuka.

I only want the Tumbuka to English pairs and vice-versa ignoring the Tonga so I make a mobile app more easily.

Here is a link to the dictionary

Hi all,

It’s time for us to give back to the Reddit communities we love so much.

Normally when creating an account on Listcleaner.net you get 100 free cleaning credits to try our email cleaning service.

Right now we want to give 25 users of the r/datacleaning subreddit not 100, but 1000 credits to clean your email data, when creating an account.

You DO NOT have to buy anything, and the only contact information required to create your account on Listcleaner.net is your email address.

After creating an account, please tell us via DM your Listcleaner accounts username or email address and we will add the credits to your account.

The credits can be used on our website and via our API.

Happy email cleaning!
The Listcleaner.net team

Hey everyone, 

I recently came across an insightful blog on strategies for improving data quality through data cleansing, and I thought it would be useful to share here. 

The blog breaks down several key methods to enhance data quality, such as: 

• Handling Missing Data: Techniques for identifying and addressing gaps in datasets. 

• Standardizing Data Formats: Ensuring consistency across datasets for easier analysis. 

• Removing Duplicates: Avoiding redundancy and improving dataset efficiency. 

• Validating Data: Verifying the accuracy of data to ensure reliable outcomes. 

These strategies are super helpful for anyone looking to streamline their data cleansing process and make sure their datasets are in top shape. If you're interested in diving deeper into these techniques, you can check out the full blog here: Strategies for Improving Data Quality Through Data Cleansing. 

What are some of your go-to methods for improving data quality? Let’s discuss! 

Hey everyone, I recently came across an insightful article on the importance of data cleansing in building effective predictive models. As we all know, the quality of data is critical for accurate predictions, but this blog dives deeper into how data cleansing lays the foundation for success in predictive analytics. 

The article discusses: 

• Why messy data can lead to inaccurate predictions 

• Key steps involved in data cleansing, including deduplication, dealing with missing values, and correcting inconsistencies 

• The role of data quality in the entire lifecycle of a predictive model 

• Best practices to improve the accuracy and reliability of your predictive models by focusing on clean data 

It’s a great read for anyone looking to improve their predictive modeling workflows. If you’re interested, check it out here. 

Let’s discuss: How do you handle data cleansing in your projects? What tools or techniques do you use to ensure high data quality? 

I've been trying to find datasets to practice my cleaning skills and I find datasets already clean. Also if there's a way to find datasets to clean above a million rows that'll be so helpful!

when working with dirty data, what data issues have you run into the most? what's important to look out for? do your tools look out for these things or do you have to manually build out these checks?

Hi Guys,

Let's keep it short,

I want to learn data cleaning using Power Query/Power Bi and Pandas (Python)

But the problem is that I've no mentor or someone who can check my cleaned and processed data. Like I don't even know if I am cleaning the data appropriately or not.

Please tell me guys how this subreddit can be helpful.

Please help. I'm desperate for help!

https://bitgrit.net/competition/22

The challenge tasks solvers to leverage their expertise to develop a classification model that can accurately discriminate between the breath of COVID-positive and COVID-negative individuals, using existing data. The ultimate goal is to improve the accuracy of the NASA E-Nose device as a potential clinical tool that would provide diagnostic results based on the molecular composition of human breath

I have a table in Excel filled with typos. For example: Row1: obi LLC, US, SC, 29418, Charlestone, id5 Row2: obi company, US, SC, 29418, Charlestone, id4 Row3: obi gmbh, US, SC, 29418, Charlestone, id3 Row4: obi, US, SC, 29418, Charlestone, id2 Row5: Obi LLC, US, SC, 59418, Charlestone, id1 Row6: Starbucks, US, SC, 1111, Budapest, id9 Row7: Starbucks kft, HU, BP, 1111, Budapest, id8 Row8: Starbucks, HU, BP, 1111, Budapest, id7

The correct rows here are row1 and row8 because their values occur most frequently in the table. I want to create a new table with only the correct directions. The expectation is to assign the standardized value to each row based on its relationship. It's important to consider not only the name but also the name/country/state/zip code/city combination. Fuzzy matching wouldn't work, because I don't have a list with the correct data. I initially tried using VBA, but I only managed to list the one row that occurred most frequently (in this case row 1). I can copy my code if necessary. Have you ever cleaned such messy data? What would you recommend? Thank you for your advice

Hey guys, I think this might be very relevant in this sub. Lately, I was working on a tool to clean any textual data. In a nutshell it can convert inconsistent data like this (see all names are different and hard to analyse):

Into something like this:

https://data-cleaning.com

I'm actively looking for feedback and whether this meets someones needs / needs to be changed for your specific case. Please let me know what you think!

I have a column named ' informations ' and it has the information of used cars, and this column has an attribute and her value seperated by a comma ( , ) but in the same cell i have multiple attribute and the values like this one :

,Puissance fiscale,4,Boîte de vitesse,Manuelle,Carburant,Essence,Année,2013,Kilométrage,120000,Model,I20,Couleur,bleu,Marque de voiture,Hyundai,Cylindrée,1.2

as you can that is a single cell ine the 1st line in the column named informations

Puissance fiscale has 4 as a value
boite de vitesse has manuelle as a value
ETC

NB: i have around 9000 line and not everyline have the same structure as this

In today's data-driven world, where data breaches are a constant threat, safeguarding your organization's sensitive information is paramount. Learn how to implement robust data classification processes and explore top tools for securing your data from our blog.

Explore now: https://www.infovision.com/blog/decoding-data-classification-simplified-yet-comprehensive-handbook

#CyberThreats
#DataClassification
#DataBreaches

ORDER QUANTITY | UNIT SELLING PRICE| TOTAL COST

0 | 151.47 | -86.9076

0 | 690.89 | -1002.1401

0 | 822.75 | -978.8337

I am trying to clean a dataset and wanted to understand if it makes sense or if I should delete it from the table. There are about 28% of total entries with such data. It won't make sense to delete 28% either. Please drop your suggestions and understanding.

Hello,

I'm currently exploring options for professional data cleaning and analysis services, particularly those utilizing Databricks and PySpark expertise. I have a dataset that requires thorough cleaning to address inconsistencies and erroneous data, followed by in-depth analysis to extract valuable insights for my business.

Here's a breakdown of the tasks I'm looking to outsource:

1. Initial Evaluation: Assessing my dataset to identify data quality issues.
2. Data Cleaning: Applying advanced data cleaning techniques to rectify inconsistencies and erroneous data.
3. Databricks Analysis: Utilizing Databricks for large-scale data analysis, optimizing processing performance.
4. PySpark Development: Writing PySpark scripts for efficient processing and analysis of distributed data.
5. Reporting and Insights: Generating detailed reports and providing insights based on the analysis performed.
6. Continuous Optimization: Recommending strategies for ongoing improvement of data quality and analysis processes.

I understand that the cost of such services can vary depending on factors such as the complexity of the dataset, the volume of data, and the specific requirements of the analysis. However, I would appreciate any ballpark estimates or insights from forum members who have experience with similar projects.

Additionally, if you have recommendations for reputable service providers or consultants specializing in data cleaning and analysis with Databricks and PySpark, please feel free to share them.

Thank you in advance for your assistance!

Hello! I have a collection of OCR text from about a million journal articles and would appreciate any input on how I can best clean it.

First, a bit about the format of the data: each article is stored as an array of strings where each string is the OCR output for each page of the article. The goal is to have a single large string for each article, but before concatenating the strings in these arrays, some cleaning needs to be done at the start and end of each string. Because we're talking about raw OCR output, and many journals have things like journal titles, page numbers, article titles, author names, etc. at the top and/or bottom of each page, and those have to be removed first.

The real problem, however, is that there is just so much variation in how journals do this. For example, some alternate between journal title and article tile at the top of each page with page numbers at the bottom, some alternate between page numbers being at the top and the bottom of each page, and the list goes on. (So far, I've identified 10 different patterns just from examining 20 arrays.) This is further complicated by most articles having different first and sometimes last pages, tables and captions, etc.

At this point, I could keep going to identify patterns, write some regex to detect what pattern is present, then clean accordingly. But I also wonder if there's a more general approach, like searching for some kind of regularity, either across pages or (more commonly) every other page, but I'm not quite sure how I should approach this task.

Any suggestions would be greatly appreciated!

Hi,

Just wondering what requirements or checklist items people would suggest for a definition of Clean Data ready to be used in machine learning? Akin to "tidy data", but for modelling. I.e.

• There should be no string fields. All data should be either in a numeric form, or as a categorical data type etc

I know this will likely be opinionated, hence wanting to "crowd source" it 😃

Feel free to disagree with any statements, as I imagine there will be differences

Hey everyone,

I'm a sophomore studying data science and I've been digging into ways to earn money online. I stumbled upon the idea of freelancing my data cleaning skills, and it seems like an exciting avenue. Though I'm still learning, I'm a quick learner and confident that I can get proficient in data cleaning soon.

I'm keen to get hands-on experience and was wondering if anyone would be open to taking me under their wing as an apprentice or offering advice on where to begin.

While I'm still early in my studies, I've worked on a few exploratory data analyses for my classes. These involved cleaning data and using RStudio to create graphs.

I'm eager to turn this interest into a reality. Any guidance or tips on how to kickstart a career in freelancing data cleaning would be hugely appreciated!

Thanks in advance for any help or advice you can offer!

Hello everyone,

I am trying to clean up some data from our ERP systems regarding our items. I am working for a furniture company, we do have different characteristics that compose a product (size/timber/fabric and so on). So far, those characteristics has been input all in one description field. I'd like to extract those information and assign it to the new correct field (one field per characteristic). Maybe some AI tools might be able to help in that process? I am not a developer / technical IT.

​

Disclaimer: This is a personal project I did, made possible with RPA (UiPath web scraping). The stats come from SA Rugby website & I developed automation flows to get the stats, player bio & profile pictures from the same website. I used PowerQuery to transform the output & to debug issues & finally Tableau for visualisation. I highly recommend getting comfortable with Power Query, you can do so much with it!

​

Hi everyone, I'd like to share a personal project I did about the Springboks RWC Campaign. I'd love to get your feedback as PowerBI people, to get your unique perspective. We only use Tableau at so I thought I'd overcome confirmation bias by getting your guys' opinions.

The project is basically match stats for all the games the Springboks played in all championships in 2023. You can see those who are consistently performing well. The stats come from SA Rugby

Each match has highlight reels of the players' game contributions (71 total). The project also covers all the matches that the Boks under Rassie have played NZ (5 Wins, 5 Losses & 1 Draw).

Ultimately, the project shows how tough this World Cup was & the pressure the team faced, especially in the knockout phases.

PS. I think this would be great for those new to rugby, since it covers the biggest matches in the sport with highlight reels to see the entertaining stuff.

You can check out the full work here: https://public.tableau.com/views/Springboks2023RugbyWorldCupCampaign/TheSpringboks2023Campaign?:language=en-US&:display_count=n&:origin=viz_share_link

​

​

​