寒玉 Blog

knime官方node梳理

Knime官方Nodes

IO

Read

  • File Reader
  • ARFF Reader
  • CSV Reader
  • Line Reader
  • Table Reader
  • PMML Reader
  • Model Reader
  • Fixed Width File Reader
  • List Files
  • Read Images
  • Explorer Browser
  • Excel Reader (XLS)
  • Read Excel Sheet Names (XLS)

File Handling

  • Remote

    • Connections

      • Create Directory
      • Delete Files
      • Download
      • Download / Upload from List
      • List Remote Files
      • Upload
      • HDFS File Permission

  • Zip

    • Unzip Files
    • Zip Files

  • Binary Objects

    • Binary Objects to Files
    • Binary Objects to PNGs
    • Binary Objects to Strings
    • Files to Binary Objects
    • PNGs to Binary Objects
    • Strings to Binary Objects

  • URI

    • Extract URI Info
    • Port to URI
    • String to URI
    • URI Port to Variable
    • URI to Port
    • URI to String
    • URL to File Path
    • URL to File Path (Variable)

  • Copy/Move Files

  • File Meta Info
  • Find MIME-Type
  • List MIME-Types

Other

  • Table Creator
  • Data Generator
  • Create Table Structure
  • Create Temp Dir
  • Send Email
  • Extract System Properties
  • Extract Context Properties

Cache

Write

  • Explorer Writer
  • CSV Writer
  • ARFF Writer
  • Table Writer
  • PMML Writer
  • Model Writer
  • Image Port Writer
  • Image Column Writer
  • Excel Sheet Appender (XLS)
  • Excel Writer (XLS)

Database

Utility

  • Database Drop Table
  • Database SQL Executor
  • SQL Extract
  • SQL Inject

Read/Write

  • Database Reader
  • Database Table Connector
  • Database Connection Table Reader
  • Database Connection Table Writer
  • Database Looping
  • Parameterized Database Query
  • Database Table Selector
  • Database Writer
  • Database Update
  • Database Delete
  • Impala Loader
  • Hive Loader

Connector

  • Impala Connector
  • Vertica Connector
  • Database Connector
  • Hive Connector
  • H2 Connector
  • Microsoft SQL Server Connector
  • MySQL Connector
  • PostgreSQL Connector
  • SQLite Connector

Manipulation

  • Database Apply-Binner
  • Database Auto-Binner
  • Database Column Rename
  • Database Numeric-Binner
  • Database Pivot
  • Database Row Filter
  • Database Query
  • Database Column Filter
  • Database Sorter
  • Database GroupBy
  • Database Joiner
  • Database Table Creator
  • Database Sampling

Structured Data

XML

  • XML Reader
  • XML Writer
  • String To XML
  • XPath
  • XSLT
  • Column To XML
  • XML Column Combiner
  • XML Row Combiner
  • XML Row Combine and Write

JSON

  • Container Input (JSON)
  • Container Output (JSON)
  • JSON Reader
  • JSON Writer
  • String to JSON
  • Table to JSON
  • JSON to Table
  • Columns to JSON
  • XML To JSON
  • JSON To XML
  • JSON Path
  • JSON Path (Dictionary)
  • JSON Column Combiner
  • JSON Row Combiner
  • JSON Row Combiner and Writer
  • JSON Transformer
  • JSON Schema Validator
  • JSON Diff

KNIME Labs

PMML Translation

  • PMML to Java

    • PMML Compiling Predictor
    • PMML Compiler
    • Compiled Model Predictor

Manipulation

Row

  • Filter

    • Filter Apply
    • Filter Apply Row Splitter
    • Filter Definition Merger
    • HiLite Row Splitter
    • Nominal Value Row Filter
    • Nominal Value Row Splitter
    • Numeric Row Splitter
    • Reference Row Filter
    • Reference Row Splitter
    • Row Filter
    • Row Splitter
    • Rule-based Row Filter
    • Rule-based Row Filter (Dictionary)
    • Rule-based Row Splitter
    • Rule-based Row Splitter (Dictionary)

  • Transform

    • Concatenate
    • Concatenate (Optional in)
    • GroupBy
    • Ungroup
    • Partitioning
    • Pivoting
    • Unpivoting
    • Rank
    • Row Sampling
    • Bootstrap Sampling
    • Equal Size Sampling
    • Shuffle
    • SorterRule-based Row Splitter (Dictionary)

  • Other

    • Rule Engine
    • Rule Engine (Dictionary)
    • Add Empty Rows
    • Extract Column Header
    • Insert Column Header
    • RowID

PMML

  • Cell To PMML
  • PMML To Cell
  • Column Filter (PMML)
  • Denormalizer (PMML)
  • Many to One (PMML)
  • Normalizer (PMML)
  • Normalizer Apply (PMML)
  • Number To String (PMML)
  • Numeric Binner (PMML)
  • One to Many (PMML)
  • String To Number (PMML)
  • XML To PMML
  • Ruleset Editor
  • Ruleset Predictor
  • Ruleset to Table

Column

  • Transform

    • Case Converter
    • Column Comparator
    • Column Resorter
    • Lag Column
    • Reference Column Resorter
    • Denormalizer
    • Extract Missing Value Cause
    • Missing Value
    • Missing Value (Apply)
    • Normalizer
    • Normalizer (Apply)
    • One to Many
    • Many to One
    • SMOTE
    • Set Operator
    • Subset Matcher

  • Binning

    • Auto-Binner
    • Auto-Binner (Apply)
    • Numeric Binner
    • Binner (Dictionary)
    • CAIM Binner
    • CAIM Applier

  • Interactive HiLite Collector

  • Table Validator
  • Table Validator (Reference)

  • Convert & Replace

    • String Manipulation
    • Category To Number
    • Category To Number (Apply)
    • Cell Replacer
    • Column Auto Type Cast
    • Column Rename
    • Column Rename (Regex)
    • Constant Value Column
    • Number To Category (Apply)
    • Number To String
    • String To Number
    • Double To Int
    • Round Double
    • String Replace (Dictionary)
    • String Replacer
    • Domain Calculator
    • Edit Numeric Domain
    • Edit Nominal Domain (Dictionary)
    • Edit Nominal Domain
    • Target Shuffling
    • Math Formula
    • Math Formula (Multi Column)

  • Filter

    • Column Filter
    • Reference Column Filter
    • Constant Value Column Filter
    • Missing Value Column Filter
    • Reference Column Splitter

  • Split & Combine

    • Cell Splitter
    • Cell Splitter By Position
    • Column Aggregator
    • Column Combiner
    • Column Merger
    • Column Splitter
    • Column Appender
    • Column to Grid
    • Create Bit Vector
    • Expand Bit Vector
    • Create Collection Column
    • Split Collection Column
    • Create Byte Vector
    • Expand Byte Vector
    • Joiner
    • Cross Joiner
    • Regex Split

Table

  • Extract Table Dimension
  • Extract Table Spec
  • Transpose

Workflow Control

Automation

  • Call Local Workflow (Row Based)
  • Container Input (Credentials)
  • Container Input (Table)
  • Container Input (Variable)
  • Container Output (Table)
  • Wait...
  • Save Workflow
  • Timer Info
  • Global Timer Info

Switches

  • Java IF (Table)
  • IF Switch
  • End IF
  • CASE Switch Data (Start)
  • CASE Switch Data (End)
  • CASE Switch Model (Start)
  • CASE Switch Model (End)
  • CASE Switch Variable (Start)
  • CASE Switch Variable (End)
  • Empty Table Switch

Loop Support

  • Breakpoint
  • Counting Loop Start
  • Chunk Loop Start
  • Column List Loop Start
  • Generic Loop Start
  • Table Row To Variable Loop Start
  • Loop End
  • Variable Condition Loop End
  • Group Loop Start
  • Interval Loop Start
  • Loop End (2 ports)
  • Loop End (Column Append)
  • Recursive Loop End
  • Recursive Loop End (2 ports)
  • Recursive Loop Start
  • Recursive Loop Start (2 ports)
  • Variable Loop End

Variables

  • Create File Name
  • Inject Variables (Data)
  • Inject Variables (Database)
  • Extract Variables (Data)
  • Extract Variables (Database)
  • Table Column to Variable
  • Table Row to Variable
  • Variable to Table Column
  • Variable to Table Row
  • Merge Variables
  • Java Edit Variable
  • Rule Engine Variable
  • Rule Engine Variable (Dictionary)
  • String Manipulation (Variable)
  • Java Edit Variable (simple)
  • Math Formula (Variable)

Meta Nodes

  • Variables Loop (Data)
  • Variables Loop (Database)
  • Iterate List of Files
  • Loop x-times

Error Handling

  • Catch Errors (DB Ports)
  • Catch Errors (Data Ports)
  • Catch Errors (Generic Ports)
  • Catch Errors (Var Ports)
  • Try (Data Ports)
  • Try (Variable Ports)
  • Active Branch Inverter

Other Data Types

Time Series

  • Meta Nodes

    • Time-Series Auto-Prediction Predictor
    • Time-Series Auto-Prediction Training
    • Seasonality Correction

  • Time Series (legacy)

    • Date Field Extractor (legacy)
    • Time Field Extractor (legacy)
    • Date/Time Shift (legacy)
    • Extract Time Window (legacy)
    • Mask Date/Time (legacy)
    • Preset Date/Time (legacy)
    • String to Date/Time (legacy)
    • Time Difference (legacy)
    • Time Generator (legacy)
    • Time to String (legacy)

  • Manipulate

    • Create Date&Time Range
    • Date & Time Difference
    • Date & Time Shift
    • Date & Time-based Row Filter
    • Modify Date
    • Modify Time
    • Modify Time Zone

  • Smoothing

    • Moving Aggregation
    • Moving Average

  • Transform

    • Date&Time to String
    • String to Date&Time
    • UNIX Timestamp to Date&Time
    • Duration to String
    • Duration to Number
    • String to Duration
    • Date&Time to legacy Date&Time
    • Legacy Date&Time to Date&Time
    • Extract Date&Time Fields
    • Extract Duration Fields
    • Window Loop Start

Scripting

Java

  • Java Snippet
  • Java Snippet (simple)
  • Java Snippet Row Filter
  • Java Snippet Row Splitter

Tools & Services

REST Web Services

  • GET Request
  • POST Request
  • PUT Request
  • DELETE Request

Apache Spark

  • Misc

    • PySpark

      • PySpark Script (1 to 1)
      • PySpark Script (1 to 2)
      • PySpark Script (2 to 1)
      • PySpark Script (2 to 2)
      • PySpark Script Source

    • Spark SQL Query

    • Java Snippet

      • Spark DataFrame Java Snippet
      • Spark DataFrame Java Snippet (Sink)
      • Spark DataFrame Java Snippet (Source)
      • Spark RDD Java Snippet
      • Spark RDD Java Snippet (Sink)
      • Spark RDD Java Snippet (Source)

    • Management

      • List Spark DataFrames/RDDs

  • Column

    • Spark Column Filter

    • Transform

      • Spark Missing Value
      • Spark Missing Value (Apply)

    • Convert & Replace

      • Spark Category To Number
      • Spark Column Rename
      • Spark Column Rename (Regex)
      • Spark Compiled Transformations Applier
      • Spark Normalizer
      • Spark Number To Category (Apply)
      • Spark Transformations Applier

    • Split & Combine

      • Spark Joiner

  • Row

    • Spark Concatenate
    • Spark GroupBy
    • Spark Partitioning
    • Spark Pivot
    • Spark Row Filter
    • Spark Row Sampling
    • Spark Sorter

  • Statistics

    • Spark Correlation Filter
    • Spark Correlation Matrix
    • Spark Linear Correlation
    • Spark Statistics

  • IO

    • Write

      • Spark to Avro
      • Spark to CSV
      • Spark to JSON
      • Spark to ORC
      • Spark to Parquet
      • Spark to Table
      • Spark to Text

    • Database

      • Database to Spark
      • Hive to Spark
      • Impala to Spark
      • Spark to Database
      • Spark to Hive
      • Spark to Impala

    • Persist Spark DataFrame/RDD

    • Unpersist Spark DataFrame/RDD

    • Read

      • Avro to Spark
      • CSV to Spark
      • JSON to Spark
      • ORC to Spark
      • Parquet to Spark
      • Table to Spark
      • Text to Spark

  • Create Big Data Test Environment

  • Create Local Big Data Environment
  • Create Spark Context (Jobserver)
  • Create Spark Context (Livy)
  • Destroy Spark Context

  • Mining

    • Scoring

      • Spark Entropy Scorer
      • Spark Numeric Scorer
      • Spark Scorer

    • Clustering

      • Spark Cluster Assigner
      • Spark k-Means

    • Spark Collaborative Filtering Learner

    • PMML

      • Spark Compiled Model Predictor
      • Spark MLlib to PMML
      • Spark PMML Model Predictor

    • Dimensionality Reduction

      • Spark PCA
      • Spark SVD

    • Prediction

      • Spark Decision Tree Learner
      • Spark Gradient-Boosted Trees Learner
      • Spark Linear Regression Learner
      • Spark Linear SVM Learner
      • Spark Logistic Regression Learner
      • Spark Naive Bayes Learner
      • Spark Predictor
      • Spark Random Forests Learner

    • Item Sets / Association Rules

      • Spark Association Rule (Apply)
      • Spark Association Rule Learner
      • Spark Frequent Item Sets

Views

Spark Line Appender

Radar Plot Appender

Property

  • Color Manager
  • Size Manager
  • Shape Manager
  • Color Appender
  • Size Appender
  • Shape Appender
  • Extract Color

Box Plot

Conditional Box Plot

HiLite Table

Histogram

Histogram (interactive)

Interactive Table

Lift Chart

Line Plot

Parallel Coordinates

Pie chart

Pie chart (interactive)

Scatter Matrix

Scatter Plot

Utility

  • Renderer to Image
  • String To SVG
  • Image To Table
  • Table To Image

Reporting

Data to Report

Image to Report

Analytics

Mining

  • Rule Induction

    • Fuzzy Rules

      • Fuzzy Rule Learner
      • Fuzzy Rule Predictor

  • Neural Network

    • MLP

      • MultiLayerPerceptron Predictor
      • RProp MLP Learner

    • PNN

      • PNN Learner (DDA)
      • PNN Predictor

  • Linear/Polynomial Regression

    • Linear Regression Learner
    • Polynomial Regression Learner
    • Regression Predictor

  • Misc Classifiers

    • K Nearest Neighbor
    • K Nearest Neighbor (Distance Function)

  • Item Sets / Association Rules

    • Association Rule Learner
    • Create Bit Vector
    • Subset Matcher

  • Scoring

    • Meta Nodes

      • Cross Validation

    • Scorer

    • Numeric Scorer
    • Entropy Scorer
    • ROC Curve
    • Enrichment Plotter

    • Cross Validation

      • X-Partitioner
      • X-Aggregator

  • Logistic Regression

    • Logistic Regression Learner
    • Logistic Regression Predictor

  • Bayes

    • Naive Bayes Learner
    • Naive Bayes Predictor

  • Feature Selection

    • Meta Nodes

      • Backward Feature Elimination
      • Forward Feature Selection

    • Feature Selection Loop Start (1:1)

    • Feature Selection Loop Start (2:2)
    • Feature Selection Loop End
    • Feature Selection Filter
    • Linear Correlation
    • Correlation Filter
    • Low Variance Filter

  • Decision Tree Ensemble

    • Random Forest

      • Random Forest Distance
      • Tree Ensemble Model Extract
      • Tree Ensemble Statistics
      • Regression
      • Classification

    • Gradient Boosting

      • Regression

        • Gradient Boosted Trees Learner (Regression)
        • Gradient Boosted Trees Predictor (Regression)
        • PMML Gradient Boosted Trees Predictor (Regression)

      • Gradient Boosted Trees to PMML

      • Classification

        • Gradient Boosted Trees Learner
        • Gradient Boosted Trees Predictor
        • PMML Gradient Boosted Trees Predictor

  • Decision Tree

    • PMML Simple Regression Tree Predictor
    • Simple Regression Tree Learner
    • Simple Regression Tree Predictor
    • Simple Regression Tree to PMML
    • Decision Tree to Ruleset
    • Decision Tree Learner
    • Decision Tree Predictor
    • Decision Tree To Image

  • PCA

    • PCA
    • PCA Compute
    • PCA Apply
    • PCA Inversion

  • Clustering

    • DBSCAN
    • k-Medoids
    • Hierarchical Clustering (DistMatrix)
    • Hierarchical Cluster View
    • Hierarchical Cluster Assigner
    • Cluster Assigner
    • Fuzzy c-Means
    • Hierarchical Clustering
    • SOTA Learner
    • SOTA Predictor
    • k-Means

  • Ensemble Learning

    • Boosting Learner Loop End
    • Boosting Learner Loop Start
    • Boosting Predictor Loop End
    • Boosting Predictor Loop Start
    • Cell To Model
    • Model Loop End
    • Model Loop Start
    • Model to Cell
    • Prediction Fusion
    • Voting Loop End

    • Meta Nodes

      • Bagging
      • Boosting Learner
      • Boosting Predictor
      • Delegating

    • PMML

      • PMML Ensemble Loop End
      • PMML Ensemble Predictor
      • PMML Ensemble to Table
      • Table to PMML Ensemble

  • MDS

    • MDS (DistMatrix)
    • MDS Projection (DistMatrix)
    • MDS
    • MDS Projection

  • SVM

    • SVM Learner
    • SVM Predictor

  • PMML

    • PMML Predictor

Distance Calculation

  • Distance Matrix

    • Distance Matrix Reader
    • Distance Matrix Writer
    • Distance Matrix Calculate
    • Distance Matrix Pair Extractor
    • Similarity Search

  • Distance Functions

    • Numeric Distances
    • String Distances
    • Bit Vector Distances
    • Byte Vector Distances
    • Mahalanobis Distance
    • Matrix Distance
    • Aggregated Distance
    • Java Distance

Statistics

  • Crosstab
  • Value Counter
  • Linear Correlation

Spark Node Demo

Spark on Hadoop

2019-06-21-18-03-17

This workflow demonstrates the usage of the Spark MLlib Decision Tree Learner and Spark Predictor. It also demonstrates the conversion of categorical columns into numerical columns which is necessary since the MLlib algorithms only support numerical features and labels.

Requirements: - KNIME Analytics Platform + KNIME Extension for Apache Spark - Hadoop Cluster e.g. Hortonworks Sandbox (http://hortonworks.com/products/hortonworks-sandbox/) + Spark Job Server needs to be installed and configured + Apache Spark 1.2.1 to 2.2 For details see https://www.knime.org/knime-spark-executor

hive to spark to hive

2019-06-21-18-17-07

This workflow demonstrates the usage of the Hive to Spark and Spark to Hive nodes that allow you to transfer data between Apache Spark and Apache Hive.

Requirements: - KNIME Analytics Platform + KNIME Big Data Connectors + KNIME extension for Apache Spark - Hadoop Cluster e.g. Hortonworks Sandbox (http://hortonworks.com/products/hortonworks-sandbox/) + Spark Job Server needs to be installed and configured + Apache Spark 1.2.1 to 2.2 For details see https://www.knime.org/knime-spark-executor and https://www.knime.org/knime-big-data-connectors

Comprehensive Model Learning Mass Prediction

2019-06-21-19-04-32

This workflow demonstrates the usage of the Spark Compiled Model Predictor node which converts a given PMML model into machine code and uses the compiled model to predict vast amounts of data in parallel within Apache Spark.

Requirements: - KNIME Analytics Platform + KNIME Ensemble Learning Wrappers + KNIME Extension for Apache Spark - Hadoop Cluster e.g. Hortonworks Sandbox (http://hortonworks.com/products/hortonworks-sandbox/) + Spark Job Server needs to be installed and configured + Apache Spark 1.2.1 to 2.2 For details see https://www.knime.org/knime-spark-executor

XMind: ZEN - Trial Version

Published

Category

bigdata

Tags