data-science

By Michael Nocito , data analyst · Published August 7, 2026 By the end of this page you can take a DataFrame, summarize it by any column or combination of columns, get several statistics at once with sensible names, and account for every row that went in, including the ones pandas would otherwise drop without telling you. It is about twenty-five minutes, and every output shown was produced by act…

By Michael Nocito , data analyst · Published August 7, 2026 This article gives you the test that catches the most common beginner mistake in pivot tables, and the ten-second fix. The test is one line: would adding two of these together mean anything? If the answer is no, the column is a label, and it must never be summed. The mistake happens without you doing anything wrong. You drag a column int…

This article introduces the evolution of an open standard designed to help enterprise data and AI tools interpret the same business metrics consistently.

Bibliometrics is increasingly being used by the knowledge community and librarians to easily analyze patterns in knowledge. In the field, the use of data from databases that provide bibliometric information is not always completely clean, so pre-processing is required. Several previous studies have shown that bibliometric analysis begins with a simple pre-processing step. The goal of this researc…

As AI research continue to accelerate, many Statistics and biostatistics faculty members are eager to explore new more data-intensive research directions. Recently, a new resource for the statistics and biostatistics community was launched: A comprehensive biomedical data resource guide, available at here. This platform serves as a curated collection of links to diverse biomedical datasets, enabl…

The benefits of having data Two ways to look at drive failures and temperature. Almost all recent articles and papers I have read on hard drive failure rates refer to either Failure Trends in a Large Disk Drive Population from Google, or Estimating Drive Reliability in Desktop Computers and Consumer Electronics Systems from Seagate. Despite both sounding and looking authoritative, these papers co…

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate , you need to understand two related but different topics: Slowly Changing Dimensions , SCD Data quality expectations in Lakeflow Spark Declarative Pipelines SCD is about how to model changes in dimension data over time. Data quality expectations are about validating records as they flow through a pipeline. T…

A collaboration between the UT San Antonio College of AI, Cyber and Computing and UTSA Athletics is turning sports data into actionable insights while giving students the opportunity to tackle real-world challenges. While Roadrunner student-athletes focus on home runs, stolen bases and preparing for game-day , data science students are analyzing everything from player performance […] The post A w…

When you're starting your Machine Learning journey, one of the first things you'll need to learn is how to load your dataset into your Jupyter Notebook. Let's learn how to do it in the simplest way. 🚀 🐍 Step 1: Import Pandas import pandas as pd Here, we're importing the Pandas library and giving it the shorter name pd. Pandas is a popular Python library used for data analysis and manipulation. It…

As Director of the Scientific Data Division at Lawrence Berkeley National Laboratory, Ana Kupresanin leads scientists and engineers who develop the methods, software, workflows, and infrastructure needed to make scientific data usable, reliable, and reusable for science and AI. The division works across the scientific data lifecycle, helping researchers organize, curate, manage, access, analyze, …

A new data-driven dashboard produced by the Cline Center for Advanced Social Research and an interdisciplinary team of University of Illinois Urbana-Champaign experts uses predictive modeling to pinpoint U.S. counties that have unusually high or low numbers of police use of lethal force incidents relative to national benchmarks.

By convening four laboratories from across the nation to test an experimental catalyst, researchers demonstrated the importance of generating highly reproducible experimental data when building AI models for investigations in science.

Notre Dame’s Data, AI, and Computing Initiative outlines a multi-year, campus-wide effort to position the University as a trusted leader in emerging technologies. The Initiative convenes discussions and action for research, education and impact across colleges, schools, centers, and disciplines in support of Notre Dame’s broader strategic framework.

Extracting structured data from invoices and contracts with one API call I've been working on a document analysis API and wanted to share a pattern that saved me from writing custom parsers for every document type my clients throw at me. The problem If you work with LATAM businesses, you know the pain: invoices in PDF (sometimes scanned), contracts in Word, receipts as phone photos. Every client …

Biology today produces more data than anyone knows what to do with. Sequencers spit out genomes, single-cell experiments generate massive expression matrices, proteomics platforms churn out thousands of protein measurements, and yet, turning all of that into an actual understanding of what is happening inside a cell or a disease remains painfully slow. A team […] The post Meet BiOmics: The AI Age…

What's the most popular number in Hacker News titles? Two consecutive titles on the HN front page yesterday had a 6 in them. This means nothing. But it’s the sort of nothing that lodges in your brain until you do something about it, so what is the most popular number in Hacker News titles? ClickHouse hosts the full HN dataset in their public playground, and I am exactly the kind of person who fin…

For a decade, the file format layer was the most settled real estate in data. Apache Parquet held the analytical world, ORC held the Hive legacy estates, and the interesting arguments all happened in the layers above. Then, in the span of about three years, the bottom of the stack became the most intellectually active corner of the industry: a research wave produced BtrBlocks, FastLanes, ALP, and…