Lecture 25: The Bigger Picture

Test¶

• Worth 20%

• Expect to do well

• The example questions are your practice test

• “This test is meant to evaluate understanding, not syntax or terminology.”

Schedule¶

Python beyond data analysis¶

We’ve been focusing on using Python and pandas for data analysis. What else is Python used for?

Data engineering¶

• Automation / recurring processes

• Copying/moving/processing/publishing data, especially Big Data

• ETL

• Monitoring/alerting

Web development¶

• Building web sites that are interactive (more than just content)

• Forms

• Presenting data

• Workflows, such as:

  • Signing up for things

  • Paying for things

• Back ends / APIs

Machine learning¶

• Statistics, but fancy

• Building models — lists of examples:

  • Hugging Face

  • TensorFlow.js

• Finding patterns

• Recommendations

• Detection

When people say “artificial intelligence,” they usually mean “machine learning.”

Source, with more thorough explanation

The process¶

High-level

1. Create a model

   1. Gather a bunch of data for training

   2. If supervised machine learning, label it (give it the right answers)

   3. Segment into training and test data

   4. Train the model against the training dataset (have it identify patterns)

   5. Test the model against the test dataset

2. Run against new data

3. If reinforcement learning, model refines itself

LLMs¶

You have a head start: The fundamentals are applicable anywhere you’re using code.

Advanced Computing for Policy¶

Ask Me Anything (AMA)¶

Thanks to the TAs!

Thank you!¶

Keep in touch.