0
Follow
0
View

AWS Step Functions EMR PySpark Task Step failed

cwhpxjx 注册会员
2023-01-24 20:34

The issue was that the EMR cluster was created without Spark option.

Solution was to add "Name": "Spark" under "Applications" property in the step functions create cluster task:

    "EMR CreateCluster": {
      "Type": "Task",
      "Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync",
      "Parameters": {
        "Name": "ExampleCluster",
        "VisibleToAllUsers": true,
        "ReleaseLabel": "emr-5.33.0",
        "Applications": [
          {
            "Name": "Hive",
            "Name": "Spark"
          }
        ],

About the Author

Question Info

Publish Time
2023-01-24 20:34
Update Time
2023-01-24 20:34