How to Run Code in Google Colab After Uploading the Files

Google Colaboratory is a gratis Jupyter notebook surroundings that runs on Google'south deject servers, letting the user leverage backend hardware like GPUs and TPUs. This lets you practise everything you tin in a Jupyter notebook hosted in your local machine, without requiring the installations and setup for hosting a notebook in your local automobile.

Colab comes with (almost) all the setup you need to start coding, but what it doesn't accept out of the box is your datasets! How do yous access your data from within Colab?

In this article we will talk almost:

How to load data to Colab from a multitude of information sources
How to write back to those information sources from within Colab
Limitations of Google Colab while working with external files

Directory and file operations in Google Colab

Since Colab lets you do everything which you can in a locally hosted Jupyter notebook, y'all tin too apply shell commands like ls, dir, pwd, cd, true cat, echo, et cetera using line-magic (%) or fustigate (!).

To browse the directory structure, y'all can use the file-explorer pane on the left.

How to upload files to and download files from Google Colab

Since a Colab notebook is hosted on Google'due south cloud servers, there's no straight access to files on your local drive (unlike a notebook hosted on your machine) or any other surroundings by default.

However, Colab provides diverse options to connect to almost whatever data source you can imagine. Allow united states of america see how.

Accessing GitHub from Google Colab

You can either clone an entire GitHub repository to your Colab surroundings or access individual files from their raw link.

Clone a GitHub repository

Y'all tin can clone a GitHub repository into your Colab environs in the same fashion as y'all would in your local automobile, using git clone. One time the repository is cloned, refresh the file-explorer to browse through its contents.

And then you can only read the files as you would in your local machine.

Load individual files direct from GitHub

In case you only have to work with a few files rather than the entire repository, you can load them directly from GitHub without needing to clone the repository to Colab.

To do this:

click on the file in the repository,
click on View Raw,
re-create the URL of the raw file,
use this URL as the location of your file.

Accessing Local File Arrangement to Google Colab

You can read from or write to your local file system either using the file-explorer, or Python code:

Admission local files through the file-explorer

Uploading files from local file system through file-explorer

You tin either use the upload option at the top of the file-explorer pane to upload whatever file(due south) from your local file system to Colab in the present working directory.

To upload files directly to a subdirectory you demand to:

1. Click on the three dots visible when you hover in a higher place the directory

2. Select the "upload" option.

3. Select the file(s) you wish to upload from the "File Upload" dialog window.

4. Look for the upload to complete. The upload progress is shown at the lesser of the file-explorer pane.

Once the upload is complete, y'all tin can read from the file as yous would normally.

Downloading files to local file organisation through file-explorer

Click on the three dots which are visible while hovering to a higher place the filename, and select the "download" selection.

Accessing local file system using Python code

This stride requires you lot to first import the files module from the google.colab library:

          from          google.colab          import          files

Uploading files from local file organisation using Python lawmaking

You lot utilise the upload method of the files object:

uploaded = files.upload()

Running this opens the File Upload dialog window:

Select the file(s) you wish to upload, then wait for the upload to complete. The upload progress is displayed:

The uploaded object is a lexicon having the filename and content every bit information technology'southward key-value pairs:

Once the upload is complete, you can either read it as any other file from colab:

df4 = pd.read_json("News_Category_Dataset_v2.json", lines=True)

Or read it directly from the uploaded dict using the io library:

          import          io df5 = pd.read_json(io.BytesIO(uploaded['News_Category_Dataset_v2.json']), lines=True)

Make sure that the filename matches the name of the file you lot wish to load.

Downloading files from Colab to local file system using Python lawmaking:

The download method of the files object can be used to download whatever file from colab to your local drive. The download progress is displayed, and once the download completes, you tin choose where to save it in your local motorcar.

Accessing Google Drive from Google Colab

You tin can use the drive module from google.colab to mount your entire Google Drive to Colab past:

1. Executing the beneath lawmaking which will provide you with an hallmark link

          from          google.colab          import          drive drive.mount('/content/gdrive')

two. Open the link

iii. Cull the Google business relationship whose Drive you lot want to mount

iv. Allow Google Drive Stream admission to your Google Account

5. Re-create the code displayed, paste it in the text box every bit shown beneath, and press Enter

Once the Drive is mounted, you'll go the bulletin "Mounted at /content/gdrive", and you lot'll be able to scan through the contents of your Drive from the file-explorer pane.

Now you tin can interact with your Google Bulldoze as if it was a folder in your Colab environs. Any changes to this folder volition reflect directly in your Google Drive. You can read the files in your Google Drive as any other file.

You can even write direct to Google Drive from Colab using the usual file/directory operations.

!touch          "/content/gdrive/My Bulldoze/sample_file.txt"

This will create a file in your Google Drive, and will be visible in the file-explorer pane in one case you refresh it:

Accessing Google Sheets from Google Colab

To access Google Sheets:

1. You demand to first authenticate the Google account to be linked with Colab by running the code below:

          from          google.colab          import          auth auth.authenticate_user()

two. Executing the higher up code will provide you with an authentication link. Open the link,

three. Choose the Google account which you want to link,

iv. Allow Google Cloud SDK to access your Google Business relationship,

5. Finally copy the code displayed and paste it in the text box shown, and hit Enter.

To interact with Google Sheets, you need to import the preinstalled gspread library. And to authorize gspread admission to your Google business relationship, you demand the GoogleCredentials method from the preinstalled oauth2client.client library:

          import          gspread          from          oauth2client.client          import          GoogleCredentials   gc = gspread.qualify(GoogleCredentials.get_application_default())

Once the above lawmaking is run, an Awarding Default Credentials (ADC) JSON file will be created in the present working directory. This contains the credentials used by gspread to access your Google account.

Once this is washed, yous can now create or load Google sheets directly from your Colab environment.

Creating/updating a Google Sheet in Colab

1. Apply the gc object'due south create method to create a workbook:

wb = gc.create('demo')

two. Once the workbook is created, y'all tin view it in sheets.google.com.

3. To write values to the workbook, first open a worksheet:

ws = gc.open('demo').sheet1

4. And so select the jail cell(due south) you lot want to write to:

5. This creates a list of cells with their index (R1C1) and value (currently blank). You tin modify the individual cells past updating their value attribute:

6. To update these cells in the worksheet, use the update_cells method:

7. The changes will at present be reflected in your Google Sheet.

Downloading data from a Google Canvass

one. Use the gc object's open method to open up a workbook:

wb = gc.open('demo')

2. And then read all the rows of a specific worksheet by using the get_all_values method:

iii. To load these to a dataframe, you tin can use the DataFrame object's from_record method:

Accessing Google Cloud Storage (GCS) from Google Colab

Yous need to accept a Google Deject Project (GCP) to use GCS. You can create and admission your GCS buckets in Colab via the preinstalled gsutil command-line utility.

1. First specify your project ID:

project_id =          '<project_ID>'

2. To access GCS, you've to authenticate your Google account:

          from          google.colab          import          auth auth.authenticate_user()

iii. Executing the above lawmaking volition provide y'all with an authentication link. Open up the link,

iv. Choose the Google account which you desire to link,

5. Allow Google Cloud SDK to admission your Google Account,

six. Finally re-create the code displayed and paste it in the text box shown, and hit Enter.

7. Then you configure gsutil to apply your projection:

!gcloud config set project {project_id}

8. You lot tin can brand a saucepan using the brand bucket (mb) control. GCP buckets must accept a universally unique name, so use the preinstalled uuid library to generate a Universally Unique ID:

          import          uuid bucket_name = f'sample-saucepan-{uuid.uuid1()}'          !gsutil mb gs://{bucket_name}

9. One time the bucket is created, you lot tin upload a file from your colab environment to it:

!gsutil cp /tmp/to_upload.txt gs://{bucket_name}/

ten. One time the upload has finished, the file will be visible in the GCS browser for your projection: https://console.cloud.google.com/storage/browser?project=<project_id>

!gsutil cp gs://{bucket_name}/{filename} {download_location}

One time the download has finished, the file volition be visible in the Colab file-explorer pane in the download location specified.

Accessing AWS S3 from Google Colab

You lot need to take an AWS account, configure IAM, and generate your access key and secret access key to exist able to access S3 from Colab. Y'all also need to install the awscli library to your colab environment:

ane. Install the awscli library

!pip install awscli

2. Once installed, configure AWS by running aws configure:

Enter your access_key and secret_access_key in the text boxes, and printing enter.

Then you tin can download any file from S3:

!aws s3 cp s3://{bucket_name} ./{download_location} --recursive --exclude          "*"          --include {filepath_on_s3}

filepath_on_s3 tin point to a single file, or lucifer multiple files using a pattern.

Yous will be notified once the download is consummate, and the downloaded file(south) will be available in the location you specified to exist used as you lot wish.

To upload a file, but reverse the source and destination arguments:

!aws s3 cp ./{upload_from} s3://{bucket_name} --recursive --exclude          "*"          --include {file_to_upload}

file_to_upload can point to a single file, or match multiple files using a blueprint.

You will be notified once the upload is consummate, and the uploaded file(s) will exist available in your S3 bucket in the binder specified: https://s3.console.aws.amazon.com/s3/buckets/{bucket_name}/{binder}/?region={region}

Accessing Kaggle datasets from Google Colab

To download datasets from Kaggle, you starting time need a Kaggle account and an API token.

1. To generate your API token, go to "My Account", then "Create New API Token".

2. Open up the kaggle.json file, and copy its contents. Information technology should be in the form of {"username":"########", "key":"################################"}.

iii. Then run the below commands in Colab:

!mkdir ~/.kaggle  !repeat          '<PASTE_CONTENTS_OF_KAGGLE_API_JSON>'          > ~/.kaggle/kaggle.json  !chmod          600          ~/.kaggle/kaggle.json   !pip install kaggle

4. One time the kaggle.json file has been created in Colab, and the Kaggle library has been installed, you can search for a dataset using

!kaggle datasets list -s {KEYWORD}

5. And then download the dataset using

!kaggle datasets download -d {DATASET Proper noun} -p /content/kaggle/

The dataset will exist downloaded and will be bachelor in the path specified (/content/kaggle/ in this case).

Accessing MySQL databases from Google Colab

i. You need to import the preinstalled sqlalchemy library to work with relational databases:

          import          sqlalchemy

2. Enter the connectedness details and create the engine:

HOSTNAME =          'ENTER_HOSTNAME'          USER =          'ENTER_USERNAME'          Password =          'ENTER_PASSWORD'          DATABASE =          'ENTER_DATABASE_NAME'          connection_string = f'mysql+pymysql://{MYSQL_USER}:{MYSQL_PASSWORD}@{MYSQL_HOSTNAME}/{MYSQL_DATABASE}'          engine = sqlalchemy.create_engine(connection_string)

3. Finally, just create the SQL query, and load the query results to a dataframe using pd.read_sql_query():

query = f"SELECT * FROM {DATABASE}.{TABLE}"          import          pandas          as          pd df = pd.read_sql_query(query, engine)

Limitations of Google Colab while working with Files

One important caveat to remember while using Colab is that the files you upload to information technology won't be bachelor forever. Colab is a temporary environs with an idle timeout of ninety minutes and an absolute timeout of 12 hours. This means that the runtime volition disconnect if it has remained idle for 90 minutes, or if it has been in use for 12 hours. On disconnection, you lose all your variables, states, installed packages, and files and will exist continued to an entirely new and clean environment on reconnecting.

As well, Colab has a disk space limitation of 108 GB, of which only 77 GB is available to the user. While this should exist enough for most tasks, keep this in mind while working with larger datasets like image or video data.

Conclusion

Google Colab is a keen tool for individuals who want to harness the power of loftier-terminate computing resources similar GPUs, without beingness restricted by their price.

In this commodity, nosotros accept gone through virtually of the ways you can supercharge your Google Colab feel by reading external files or information in Google Colab and writing from Google Colab to those external data sources.

Depending on your use-case, or how your information compages is prepare-up, you can hands apply the to a higher place-mentioned methods to connect your data source directly to Colab, and start coding!

Other resources

Getting Started with Google CoLab | How to use Google Colab
External information: Local Files, Drive, Sheets and Cloud Storage
Importing Data to Google Colab — the Clean Way
Get Started: three Ways to Load CSV files into Colab | by A Apte
Downloading Datasets into Google Drive via Google Colab | by Kevin Luk

Currently working as a Data Scientist with Reuters helping their Editorial, Marketing, and Sales teams derive insights from information, he strongly believes that the best fashion to learn is to teach. Knowledge multiplies when shared 🙂

Follow me on

READ Next

How to Use Google Colab for Deep Learning – Complete Tutorial

9 mins read | Author Harshit Dwivedi | Updated June eighth, 2021

If yous're a programmer, yous want to explore deep learning, and need a platform to help yous do it – this tutorial is exactly for you.

Google Colab is a dandy platform for deep learning enthusiasts, and information technology tin also be used to test basic automobile learning models, gain feel, and develop an intuition about deep learning aspects such as hyperparameter tuning, preprocessing information, model complexity, overfitting and more than.

Let'south explore!

Introduction

Colaboratory by Google (Google Colab in curt) is a Jupyter notebook based runtime environment which allows you to run code entirely on the cloud.

This is necessary because it means that y'all can train large scale ML and DL models even if you don't have access to a powerful machine or a high speed internet access.

Google Colab supports both GPU and TPU instances, which makes it a perfect tool for deep learning and data analytics enthusiasts because of computational limitations on local machines.

Since a Colab notebook can be accessed remotely from any machine through a browser, it's well suited for commercial purposes besides.

In this tutorial you will learn:

Getting around in Google Colab
Installing python libraries in Colab
Downloading large datasets in Colab
Training a Deep learning model in Colab
Using TensorBoard in Colab

Continue reading ->

davisaccamented01.blogspot.com

Source: https://neptune.ai/blog/google-colab-dealing-with-files#:~:text=You%20can%20either%20use%20the,in%20the%20present%20working%20directory.&text=2.,Select%20the%20%E2%80%9Cupload%E2%80%9D%20option.