Data in Trees

Structuring and Navigating Files

Concept Acquisition

1. Data is stored on a computer disk in files.
2. Files are stored in directories (or folders).
3. Files and directories can be structured using a tree.
4. Each file’s location is defined by its path.

Tool Acquisition

1. The structure of relative and absolute paths.
2. UNIX commands
   • pwd
   • ls
   • cd
   • cat

Concept Application

1. Given a file system tree,
   1. write down the absolute path to a file.
   2. given two file system items, write the relative path from one to the other.
2. Given a set of file paths, reconstruct the file system tree.
3. Navigate a file system at the command line using the provided UNIX commands.

Gather up the stack of index cards from your students with answers to the questions asked in lecture.

On the board sketch a very simple version of the Question > Data Product diagram seen in the first lecture: Question: what were the songs of summer for Stat 133 students? Then for data, draw a box for each individual index card. Then draw a big empty “computational” box, and then ask what sort of data products could answer this question. Students may suggest different types of visualizations (e.g.) barchart ordered in descending order by popularity), summary statistics (most popular song of Stat 133 students), estimates (most popular song of Berkeley students).

Today we’ll focus on how to store that data on a computer. Draw one box inside the big “computational” box labelled “Storage”.

Terminology

File

A named container that stores data – such as text, images, or programs – on a computer disk. The file name usually ends in an extension (.txt, .png, .pdf) that indicates the format.

Example

127.txt

Golden Klaxons
3000?

One way to store the data on the index card is as a text file. Read more about this particular format here: https://en.wikipedia.org/wiki/Text_file

Questions you could ask after showing the .txt:

1. Where did the number 127 come from? It was an arbitrary ID.
2. Does anyone see anything that might be a challenge as we try to analysis this txt file as data? We will likely have to remove the ? to treat 3000 as a number.
3. Does anyone know this song? Probably not, because it seems to be a mix of a song and an artist … see next slide.

I’m not sure what happened here! I can’t find a song called “Golden Klaxons”, only this one. The question for the students - how should we handle this? Should we fix it?

(we’ll leave it for now and fix it in a later class)

Directory (or folder)

A named container that stores a set of files or other folders.

How should we organize our cards into directories?

Boardwork.

Lead students into thinking about the information that we have on the other side of the card: Lab #, Years at Cal, and Stats Major (yes/no).

As they propose one form of sorting, say by years at Cal, ask them if we can do any sorting within each directory – in this case, by Stats Major.

Whichever sorting scheme they propose, draw it on the board as a tree with a root node (an empty circle), with lines leading to each of the top level directory (say, circles labelled “One”, “Two”, “Three”, and “Four”), and then for each, lines leading to circles of the subdirectories (“S” for stats major, “NS” for non-stats major, say), then with lines leading to the files. For those, tape the index cards in the appropriate place.

How can we specify the location of a file/directory in the file system?

Path

A description of a file or directory’s location in the file system, written as a sequence of folder names that leads to it, separated by /.

Absolute path: Starts from the root of the file system, indicated by /. Example: /Documents/Schoolwork/HW1.pdf

Ask: What absolute paths are found in our tree?

Return to the board and circle a file or two and ask for someone to provide you the absolute path. Write it on the board (be sure to start with /). Repeat by asking the path to a directory. Circle that directory and write the path next to it (it will start with a / and end with the name of that directory).

How can we specify the location of a file/directory in the file system?

Path

A description of a file or directory’s location in the file system, written as a sequence of folder names that leads to it, separated by /.

Absolute path: Starts from the root of the file system, indicated by /. Example: /Documents/Schoolwork/HW1.pdf

Relative path: Starts from the current directory and shows how to reach a target. .. means “go up one level”. Example if you’re in Documents, Schoolwork/HW1.pdf

Ask: What relative paths are found in our tree?

On the board pick at least two relative paths for them to give you, one that goes from a directory to a file in a subdirectory, and another that requires going “up one level” to a file in a differet directory at the same level. You can indiated the “start” and “end” of the relative path using, say a square and a triangle.

I’m getting tired of always typing the trunk of the tree…

Home directory

A special designated directory in a file system that serves as a starting point for commonly-used paths.

Written as ~, a shorthand for its absolute path.

On the tree, designate one of the nodes as the home by drawing a house around it. Then ask:

What is the absolute path of this file?

Demo: Stat DataHub

Don’t have students open their laptop, first have them just watch you.

Before class, click on the link to go to DataHub with our practice sample of index card files.

https://stat.datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fberkeley-stat133%2Fdata-in-trees&urlpath=tree%2Fdata-in-trees%2F&branch=main

1. First use the GUI to show how the file system looks similar to Finder on Mac or Windows or Google Drive - it is structured in a tree.
2. Then click File > New > Terminal to introduce them to the command line / Terminal.
3. Describe it as a way to write and record the way that we navigate a file system. We’ll soon be using these paths in computer code to navigate and manipulate files much faster and more predictably than with a GUI.
4. Tell them now you’ll show them four very useful commands to use at the command line.

Iterate back and forth between the following slides and the DataHub as you demonstrate different ideas.

The Command Line (aka Terminal)

Getting around a file system

We will need to . . .

1. Know where we are
2. Know what’s around us
3. Change where we are
4. See what’s in a file

. . . all without pointing and clicking.

UNIX¹ Commands

pwd

Returns your present working directory.

ls

Lists the contents of your present working directory.

1. UNIX is a family of operating systems that includes Linux and MacOS. Most of the commands work on windows too.

cd

Changes your directory to the path provided (after a space). Examples:

• cd Homework: (relative) only takes you into the Homework directory if you’re in Documents.
• cd /Document/Homework: (absolute) takes you into the Homework directory regardless of your pwd.

cat

Prints to the screen the contents of the file at the path provided. Example: cat 127.txt.

UNIX Commands

1. Know where we are: pwd
2. Know what’s around us: ls
3. Change where we are: cd
4. See what’s in a file cat

There are many more, like . . .

• Make a new directory: mkdir
• Copy a file: cp
• Remove a file: rm

Practice: Stat DataHub

Practice on a sample of our index cards on a cloud-hosted machine that has a directory containing sample of our index card files.

Login Here:

https://stat.datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fberkeley-stat133%2Fdata-in-trees&urlpath=tree%2Fdata-in-trees%2F&branch=main

Only do this if you have > 10 min left. Have the students get into groups of two and only one of them open up their laptop.

Practice

10:00

Using only the command line (not the file browser).

1. What are the names of the files stored in ~/data-in-trees/red/stat/101/1/?
2. What is the absolute path of the file named 014.txt?
3. What is the relative path of the file named 014.txt from ~/data-in-trees/red/non-stat?
4. What is the relative path of the directory at ~/data-in-trees/green/stat/ from ~/data-in-trees/red/non-stat?
5. What is the song of the summer in ~/data-in-trees/red/non-stat/101/1/003.txt?

Solutions

Solution 1

What are the names of the files stored in ~/data-in-trees/red/stat/101/1/?¹

One way is to move into the directory, then list its contents:

cd ~/data-in-trees/red/stat/101/1/
ls

012.txt  014.txt

Or list the contents without moving, by giving ls the path directly:

ls ~/data-in-trees/red/stat/101/1/

012.txt  014.txt

1. The trailing / isn’t strictly necessary here, but it’s good practice as it signals that the path points to a directory rather than a file.

Solution 2

What is the absolute path of the file named 014.txt?

An absolute path starts from the root /. Searching the tree, 014.txt lives inside red/stat/101/1/, so relative to home it is:

~/data-in-trees/red/stat/101/1/014.txt

Remember ~ is shorthand for home’s absolute path. On DataHub home is /home/jovyan, so the fully-written absolute path is:

/home/jovyan/data-in-trees/red/stat/101/1/014.txt

You can confirm the home path by running pwd after cd ~.

Solution 3

What is the relative path of the file named 014.txt from ~/data-in-trees/red/non-stat?

From red/non-stat, step up one level (..) to red, then descend into stat/101/1/:

../stat/101/1/014.txt

Check it from that starting directory by printing the file at the relative path:

cat ../stat/101/1/014.txt

lebi tirar mas fotos
5200

Solution 4

What is the relative path of the directory at ~/data-in-trees/green/stat/ from ~/data-in-trees/red/non-stat?

From red/non-stat, step up two levels: .. reaches red, and ../.. reaches data-in-trees. Then descend into green/stat:

../../green/stat

The two colors (red, green) sit at the same level, so any path between them must first climb up to their shared parent, data-in-trees.

Check it by walking the path yourself. First confirm where you are:

pwd

/home/jovyan/data-in-trees/red/non-stat

Then follow the relative path and confirm you arrived:

cd ../../green/stat
pwd

/home/jovyan/data-in-trees/green/stat

Solution 5

What is the song of the summer in ~/data-in-trees/red/non-stat/101/1/003.txt?

Print the file’s contents with cat:

cat ~/data-in-trees/red/non-stat/101/1/003.txt

which shows:

lebi tirar mas fotos
5200

So the song is lebi tirar mas fotos (the second line, 5200, is the accompanying count).

Alternatively, cd into the file’s directory first, then cat just the bare file name:

cd ~/data-in-trees/red/non-stat/101/1/
cat 003.txt

lebi tirar mas fotos
5200

For Next Time

1. Fill out the Stat 133 Welcome Survey
2. Work on the first questions from PS 1