tensorflow - s3 object does not exist

The solution was not to specify credentials (in such case it uses the role attached to the machine), but instead just specify the region name (for some reason it didn't read it from ~/.aws/config file) as follows:

import boto3
import os

session = boto3.Session()
os.environ['AWS_REGION'] = session.region_name

Suggestion : 2

I had the same problem when trying to anycodings_amazon-s3 access files in private S3 bucket from anycodings_amazon-s3 Sagemaker notebook. The mistake I made anycodings_amazon-s3 was to try using credentials I obtained anycodings_amazon-s3 from boto3, which seem not to be valid anycodings_amazon-s3 outside.,I have awscli set up with all credentials, anycodings_tensorflow and boto3 can view and download the file in anycodings_tensorflow question. I am wondering how I can get anycodings_tensorflow Tensorflow to have S3 access directly when anycodings_tensorflow the bucket is private.,How to install python 2.7.13 with win-32?? Using anaconda jupyter-notebook, creating a new enviroment,However, the same line on a public object anycodings_tensorflow works without an error: print anycodings_tensorflow file_io.stat('s3://ryft-public-sample-data/wikipedia-20150518.bin')

The solution was not to specify anycodings_amazon-s3 credentials (in such case it uses the anycodings_amazon-s3 role attached to the machine), but anycodings_amazon-s3 instead just specify the region name anycodings_amazon-s3 (for some reason it didn't read it from anycodings_amazon-s3 ~/.aws/config file) as follows:

import boto3
import os

session = boto3.Session()
os.environ['AWS_REGION'] = session.region_name

Suggestion : 3

1. Make sure your script can handle --model_dir as an additional command line argument. If you did not specify a location when the TensorFlow estimator is constructed a S3 location under the default training job bucket will be passed in here. Distributed training with parameter servers requires you use the tf.estimator.train_and_evaluate API and a S3 location is needed as the model directory during training. Here is an example:,model_dir (str) Location where model data, checkpoint data, and TensorBoard checkpoints should be saved during training. If not specified a S3 location will be generated under the training job’s default bucket. And model_dir will be passed in your training script as one of the command line arguments.,model_dir (str) Location where model data, checkpoint data, and TensorBoard checkpoints should be saved during training. If not specified a S3 location will be generated under the training job’s default bucket. And model_dir will be passed in your training script as one of the command line arguments. ,If you have a TensorFlow training script that runs outside of SageMaker please follow the directions here:

import argparse
import os

if __name__ == '__main__':

   parser = argparse.ArgumentParser()

# hyperparameters sent by the client are passed as command - line arguments to the script.
parser.add_argument('--epochs', type = int,
   default = 10)
parser.add_argument('--batch_size', type = int,
   default = 100)
parser.add_argument('--learning_rate', type = float,
   default = 0.1)

# input data and model directories
parser.add_argument('--model_dir', type = str)
parser.add_argument('--train', type = str,
   default = os.environ.get('SM_CHANNEL_TRAIN'))
parser.add_argument('--test', type = str,
   default = os.environ.get('SM_CHANNEL_TEST'))

args, _ = parser.parse_known_args()

#...load from args.train and args.test, train a model, write model to args.model_dir.

estimator = tf.estimator.Estimator(model_fn = my_model_fn, model_dir = args.model_dir)
   ...
   train_spec = tf.estimator.TrainSpec(train_input_fn, max_steps = 1000)
eval_spec = tf.estimator.EvalSpec(eval_input_fn)
tf.estimator.train_and_evaluate(mnist_classifier, train_spec, eval_spec)

estimator.fit({
   'train': 's3://my-bucket/my-training-data',
   'eval': 's3://my-bucket/my-evaluation-data'
})

parser = argparse.ArgumentParser()
parser.add_argument('--train', type = str,
   default = os.environ.get('SM_CHANNEL_TRAIN'))
parser.add_argument('--eval', type = str,
   default = os.environ.get('SM_CHANNEL_EVAL'))

from sagemaker.tensorflow
import TensorFlow

tf_estimator = TensorFlow(entry_point = 'tf-train.py', role = 'SageMakerRole',
   train_instance_count = 1, train_instance_type = 'ml.p2.xlarge',
   framework_version = '1.12', py_version = 'py3')
tf_estimator.fit('s3://bucket/path/to/training/data')

tf_estimator.fit(your_input_data, wait = False)
training_job_name = tf_estimator.latest_training_job.name

# after some time, or in a separate Python notebook, we can attach to it again.

tf_estimator = TensorFlow.attach(training_job_name = training_job_name)

Suggestion : 4

The model artifacts must be in an S3 bucket that is in the same region as the model or endpoint you are creating.,The model artifacts must be in an S3 bucket that is in the same region as the model package.,The model artifacts must be in an S3 bucket that is in the same region as the algorithm.,Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

import boto3

client = boto3.client('sagemaker')

response = client.add_association(
   SourceArn = 'string',
   DestinationArn = 'string',
   AssociationType = 'ContributedTo' | 'AssociatedWith' | 'DerivedFrom' | 'Produced'
)

{
   'SourceArn': 'string',
   'DestinationArn': 'string'
}

response = client.add_tags(
   ResourceArn = 'string',
   Tags = [{
      'Key': 'string',
      'Value': 'string'
   }, ]
)

{
   'Tags': [{
      'Key': 'string',
      'Value': 'string'
   }, ]
}

response = client.associate_trial_component(
   TrialComponentName = 'string',
   TrialName = 'string'
)