Remove Duplicate Lines - VisualCron

Please note that VisualCron support is not actively monitoring this community forum. Please use our contact page for contacting the VisualCron support directly.

Welcome Guest! To enable all features please Login or Register.

Danny van Oijen
Paid support Topic Starter

2021-06-02T11:13:48Z

Hi there,

Then i have an output like this:

R
R
R
R
ST
R
R
RP

I would like the output to be

R
ST
RP

Is this possible within vCron?
Now i use Powershell to do this but Powershell is slow....

Sponsor

Forum information

Wanna join the discussion?! Login to your Forum forum account or Register a new forum account

Gary_W
Free support

2021-06-02T21:07:34Z

Can the source of the output make it unique before supplying the output? How large is the file you are processing? How are you doing it in Powershell? Can you supply some real data?

Edited by user 2021-06-02T21:09:15Z | Reason: Not specified

Danny van Oijen
Paid support Topic Starter

2021-06-03T06:10:53Z

Gary,

Well this is it....
It is not even a file, it is the output of another job (Read Node xpath expression).

thomas
Free support

2021-06-03T12:15:46Z

I think .net is the only solution to this problem, aside from powershell. Something like this should work (System and System.Linq references need to be added)

Edited by user 2021-06-03T12:17:02Z | Reason: Not specified

Danny van Oijen
Paid support Topic Starter

2021-06-03T12:30:42Z

Thomas,

Well..... that is the next issue :D
I never used .net in vcron before.
I get this error:

thomas
Free support

2021-06-03T12:50:31Z

Steps below. I rewrote the code a bit. VC didn't like .Distinct for some reason, so I changed the logic to avoid Linq.

1) Add a .Net code exectute task

2) Edit references

3) Add the two references in yellow

4) Add this code . Make sure you add the using statements at the top

5) Click compile

6) Click refresh methods

7) Pass in the output from the previous task

Copy paste the code below:

using System;
using System.Collections.Generic;

public class StringUtils
{

    public static string GetDistinct(string input)
    {
        string[] splitByLine = input.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);
        Dictionary<string, string> uniqueItems = new Dictionary<string, string>();
        foreach (var item in splitByLine)
        {
            if (!uniqueItems.ContainsKey(item))
            {
                uniqueItems.Add(item, item);
            }
        }
        return string.Join(Environment.NewLine, uniqueItems.Values);
    }
}

Edited by user 2021-06-03T12:53:30Z | Reason: Not specified

Danny van Oijen
Paid support Topic Starter

2021-06-03T13:05:55Z

Thomas,

It worked thanks!!
This is much faster than PowerShell.

Gary_W
Free support

2021-06-04T03:26:16Z

For the fun of it would you mind trying this powershell version of Thomas' code? I felt like a challenge that's all :-)
string_in is a parameter that is equal to {TASK(PrevTask|StdOut)}. I'm interested in performance compared to the .net version just for comparison's sake. You didn't tell us how many rows you are dealing with so I couldn't perform a valid test but this works for your sample data anyway in 1.2 seconds. Thank you.

Param(
  [Parameter(Mandatory=$true)]
  [string] $string_in
)
$splitByLine = $string_in.Split()
$uniqueItems = @{}

Foreach($i in $splitByLine )
{
  if (!$uniqueItems.ContainsKey($i))
  {
    $uniqueItems.Add($i, $i);
  }
}
$uniqueItems.Keys.ForEach({"$_ $($uniqueItems.key)"})

thomas
Free support

2021-06-04T06:01:16Z

I just tested it for "fun" 🙂 The sql fetches 10000 rows where i just grap the first letter of some string. Both .net and powershell finish in less than a second, so I assume the original powershell code was inefficient in some way

Danny van Oijen
Paid support Topic Starter

2021-06-04T06:39:29Z

#10

Thomas, Gary,

The code i wrote works also that fast but that was not the issue (maybe i had to be more clear, sorry), issue i had was that just starting powershell every time takes a bit more time than .net.
You have to imagine that the job runs in a loop for let's say 500 times.
So if starting PowerShell takes 1,5 seconds longer than the .net version than you see it takes considderable more time.

And yes i know it works but i like to create a carnival festival of vcron also :D
Just the looks of my collegues when vcron starts and the green bars are flickering like hell.