Field promotion

By default, prefer field promotion. Field promotion avoids hotspotting in almost all cases, and it tends to make
it easier to design a row key that facilitates queries

Storage, queries, and streaming inserts

Explanation
Google charges for storage, queries, and streaming inserts. Loading data from a file and exporting data are free
operations.

Hive

Pig

Spark

Workers and parameter servers

The CUSTOM tier is not a set tier, but rather enables you to use your own cluster specification. When you use
this tier, set values to configure your processing cluster according to these guidelines:
You must set TrainingInput.masterType to specify the type of machine to use for your master node.
You may set TrainingInput.workerCount to specify the number of workers to use.
You may set TrainingInput.parameterServerCount to specify the number of parameter servers to use.
You can specify the type of machine for the master node, but you can't specify more than one master node

To make sure your model is generalized for more than just the training data

The flaw with evaluating a predictive model on training data is that it does not inform you on how well the
model has generalized to new unseen data. A model that is selected for its accuracy on the training dataset
rather than its accuracy on an unseen test dataset is very likely to have lower accuracy on an unseen test
dataset. The reason is that the model is not as generalized. It has specialized to the structure in the training
dataset. This is called overfitting.

You need to integrate with Google BigQuery.

Explanation
For example, if you plan to store extensive historical data for a large number of remote-sensing devices and
then use the data to generate daily reports, the cost savings for HDD storage may justify the performance
tradeoff. On the other hand, if you plan to use the data to display a real-time dashboard, it probably would not
make sense to use HDD storage—reads would be much more frequent in this case, and reads are much slower
with HDD storage.

The wide model is used for memorization, while the deep model is used for generalization.

A good use for the wide and deep model is a recommender system.

Explanation
Can we teach computers to learn like humans do, by combining the power of memorization and
generalization? It's not an easy question to answer, but by jointly training a wide linear model (for
memorization) alongside a deep neural network (for generalization), one can combine the strengths of both to
bring us one step closer. At Google, we call it Wide & Deep Learning. It's useful for generic large-scale
regression and classification problems with sparse inputs (categorical features with a large number of possible
feature values), such as recommender systems, search, and ranking problems.

To give a user access to only one table in a project, grant the user the Bigtable Editor role for that table.

Explanation
For Cloud Bigtable, you can configure access control at the project level. For example, you can grant the
ability to:
Read from, but not write to, any table within the project.
Read from and write to any table within the project, but not manage instances.
Read from and write to any table within the project, and manage instances

TensorFlow

Explanation
Cloud ML Engine mainly does two things:
Enables you to train machine learning models at scale by running TensorFlow training applications in the
cloud.
Hosts those trained models for you in the cloud so that you can use them to get predictions
about new data.

Both batch and streaming

When you apply a BigQueryIO.Write transform in batch mode to write to a single table, Dataflow invokes a
BigQuery load job. When you apply a BigQueryIO.Write transform in streaming mode or in batch mode using
a function to specify the destination table, Dataflow uses BigQuery's streaming inserts

Give a user access to view all datasets in a project, but not run queries on them.

Primitive roles can be used to give owner, editor, or viewer access to a user or group, but they can't be used to
separate data access permissions from job-running permissions

Splitting tables into multiple tables; putting data in partitions

If you split a table into multiple tables (such as one table for each day), then you can limit your query to the
data in specific tables (such as for particular days). A better method is to use a partitioned table, as long as
your data can be separated by the day.