Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add interface #33

Closed
wants to merge 3 commits into from
Closed

Add interface #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9c976eb
Improve experiments
Vitor-Git15 Nov 10, 2023
d7c6f47
Improved Lushu test measurements
Vitor-Git15 Nov 10, 2023
fcdd7c0
Adding implementation of the Interface, basic functions, and tutorial…
Vitor-Git15 Nov 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Lushu/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ fun newJarTask(
): Task {
return task(name, type = Jar::class) {
description = desc
archiveBaseName.set(name.substring(0, name.length - 3))
manifest {
attributes(manifestAttributes)
}
Expand Down
20 changes: 18 additions & 2 deletions Lushu/src/main/kotlin/lushu/TestApps/StressTest/Context/WithLushu/App.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@ package lushu.TestApps.StressTest.Context.WithLushu
import lushu.ContextGrammar.Grammar.Grammar
import lushu.Merger.Lattice.Node.MergerS
import java.io.File
import kotlin.system.measureTimeMillis

fun main(args: Array<String>) {
val runtime = Runtime.getRuntime()

if (args.size < 4) {
println(
"Usage: <this-program> <merger-config-file>\n" +
Expand All @@ -20,6 +23,19 @@ fun main(args: Array<String>) {

MergerS.load(configFilePath)
var grammar: Grammar = Grammar()
grammar.trainMap(File(patternsFilePath))
grammar.testMap(testFilePath, outputFilePath)

if (!patternsFilePath.isNullOrBlank()) {
var file = File(patternsFilePath)
grammar.trainMap(file)
}

var time: Long = 0
time = measureTimeMillis {

grammar.testMap(testFilePath, outputFilePath)
}
val totalMemory = runtime.totalMemory()
val usedMemory = totalMemory - runtime.freeMemory()
System.err.println("$time")
System.err.println("$usedMemory")
}
7 changes: 7 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,13 @@ Here are some more examples of words you can input:
Also, try specifying different YAML configuration files. You may find it easier
to edit the example file in `./example/config.yaml`.


### Video Tutorial

This page describes the basic steps to run Lushu. For a more detailed description of the process,
take a look into this
[video](https://youtu.be/s17i2BhI_Eo)
Comment on lines +117 to +118
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
take a look into this
[video](https://youtu.be/s17i2BhI_Eo)
check out our [Tutorial Video](https://youtu.be/s17i2BhI_Eo).


## Testing

To test, run `gradle test`. Find all source code for the tests under
Expand Down
57 changes: 38 additions & 19 deletions test/context_tests/compare_to_beautifulsoup/analysis.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

Binary file removed test/context_tests/compare_to_beautifulsoup/stress_test/StressTestMemory.png
View file
Open in desktop
Binary file not shown.
Binary file added test/context_tests/compare_to_beautifulsoup/stress_test/StressTestMemoryBS.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed test/context_tests/compare_to_beautifulsoup/stress_test/StressTestTime.png
View file
Open in desktop
Binary file not shown.
Binary file added test/context_tests/compare_to_beautifulsoup/stress_test/StressTestTimeBS.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
95 changes: 66 additions & 29 deletions test/context_tests/compare_to_comby/analysis.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion test/context_tests/compare_to_comby/max_blocks/Comby/10000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +0,0 @@
653 matches
1 change: 0 additions & 1 deletion test/context_tests/compare_to_comby/max_blocks/Comby/100000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +0,0 @@
7000 matches
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/1.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/10.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/100.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
4
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/1000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
79
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/10000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
653
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/100000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
7000
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/1000000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
71931
1 change: 1 addition & 0 deletions test/context_tests/compare_to_comby/max_blocks/Lushu(JVM)/10000000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
721701
18 changes: 9 additions & 9 deletions test/context_tests/compare_to_comby/results/Comby/1-0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 1 test/context_tests/compare_to_comby/max_blocks/Comby/1.txt 1000 grep maximum"
User time (seconds): 0.09
System time (seconds): 0.03
Percent of CPU this job got: 93%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.13
User time (seconds): 0.10
System time (seconds): 0.06
Percent of CPU this job got: 92%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.17
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 49936
Maximum resident set size (kbytes): 49800
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 11
Minor (reclaiming a frame) page faults: 16512
Voluntary context switches: 14
Involuntary context switches: 5
Major (requiring I/O) page faults: 16
Minor (reclaiming a frame) page faults: 16530
Voluntary context switches: 17
Involuntary context switches: 8
Swaps: 0
File system inputs: 0
File system outputs: 32
Expand Down
16 changes: 8 additions & 8 deletions test/context_tests/compare_to_comby/results/Comby/10-0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 10 test/context_tests/compare_to_comby/max_blocks/Comby/10.txt 1000 grep maximum"
User time (seconds): 0.09
User time (seconds): 0.10
System time (seconds): 0.02
Percent of CPU this job got: 88%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.13
Percent of CPU this job got: 85%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.14
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 49672
Maximum resident set size (kbytes): 49772
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 131
Minor (reclaiming a frame) page faults: 16440
Voluntary context switches: 94
Involuntary context switches: 1
Major (requiring I/O) page faults: 113
Minor (reclaiming a frame) page faults: 16420
Voluntary context switches: 78
Involuntary context switches: 3
Swaps: 0
File system inputs: 0
File system outputs: 32
Expand Down
16 changes: 8 additions & 8 deletions test/context_tests/compare_to_comby/results/Comby/100-0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 100 test/context_tests/compare_to_comby/max_blocks/Comby/100.txt 1000 grep maximum"
User time (seconds): 0.14
System time (seconds): 0.02
Percent of CPU this job got: 78%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.21
System time (seconds): 0.04
Percent of CPU this job got: 80%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.23
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 49964
Maximum resident set size (kbytes): 49924
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 101
Minor (reclaiming a frame) page faults: 16543
Voluntary context switches: 67
Involuntary context switches: 2
Major (requiring I/O) page faults: 68
Minor (reclaiming a frame) page faults: 16564
Voluntary context switches: 46
Involuntary context switches: 7
Swaps: 0
File system inputs: 0
File system outputs: 40
Expand Down
16 changes: 8 additions & 8 deletions test/context_tests/compare_to_comby/results/Comby/1000-0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 1000 test/context_tests/compare_to_comby/max_blocks/Comby/1000.txt 1000 grep maximum"
User time (seconds): 3.07
System time (seconds): 0.04
User time (seconds): 3.11
System time (seconds): 0.02
Percent of CPU this job got: 95%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.27
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.28
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 67696
Maximum resident set size (kbytes): 67768
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 242
Minor (reclaiming a frame) page faults: 20807
Voluntary context switches: 198
Involuntary context switches: 18
Major (requiring I/O) page faults: 104
Minor (reclaiming a frame) page faults: 20968
Voluntary context switches: 70
Involuntary context switches: 19
Swaps: 0
File system inputs: 0
File system outputs: 40
Expand Down
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/10000-0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 10000 test/context_tests/compare_to_comby/max_blocks/Comby/10000.txt 1000 grep maximum"
User time (seconds): 26.44
System time (seconds): 0.16
Percent of CPU this job got: 97%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:27.16
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 73404
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 22412
Voluntary context switches: 8
Involuntary context switches: 194
Swaps: 0
File system inputs: 0
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/10000-1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 10000 test/context_tests/compare_to_comby/max_blocks/Comby/10000.txt 1000 grep maximum"
User time (seconds): 24.93
System time (seconds): 0.23
Percent of CPU this job got: 97%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.74
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 73276
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 142
Minor (reclaiming a frame) page faults: 22373
Voluntary context switches: 118
Involuntary context switches: 272
Swaps: 0
File system inputs: 0
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/10000-2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 10000 test/context_tests/compare_to_comby/max_blocks/Comby/10000.txt 1000 grep maximum"
User time (seconds): 29.99
System time (seconds): 0.06
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:30.65
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 73372
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 121
Minor (reclaiming a frame) page faults: 22344
Voluntary context switches: 95
Involuntary context switches: 193
Swaps: 0
File system inputs: 0
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/10000-3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 10000 test/context_tests/compare_to_comby/max_blocks/Comby/10000.txt 1000 grep maximum"
User time (seconds): 28.26
System time (seconds): 0.12
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:28.94
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 73460
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 153
Minor (reclaiming a frame) page faults: 22295
Voluntary context switches: 114
Involuntary context switches: 223
Swaps: 0
File system inputs: 0
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/10000-4
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 10000 test/context_tests/compare_to_comby/max_blocks/Comby/10000.txt 1000 grep maximum"
User time (seconds): 31.69
System time (seconds): 0.04
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.33
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 73328
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 136
Minor (reclaiming a frame) page faults: 22272
Voluntary context switches: 101
Involuntary context switches: 139
Swaps: 0
File system inputs: 0
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/100000-0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 100000 test/context_tests/compare_to_comby/max_blocks/Comby/100000.txt 1000 grep maximum"
User time (seconds): 444.05
System time (seconds): 0.09
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 7:32.41
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 161008
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 8
Minor (reclaiming a frame) page faults: 44520
Voluntary context switches: 13
Involuntary context switches: 2590
Swaps: 0
File system inputs: 0
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/100000-1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 100000 test/context_tests/compare_to_comby/max_blocks/Comby/100000.txt 1000 grep maximum"
User time (seconds): 443.71
System time (seconds): 0.11
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 7:32.51
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 160912
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 378
Minor (reclaiming a frame) page faults: 44161
Voluntary context switches: 346
Involuntary context switches: 2955
Swaps: 0
File system inputs: 14672
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
23 changes: 0 additions & 23 deletions test/context_tests/compare_to_comby/results/Comby/100000-2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +0,0 @@
Command being timed: "bash test/context_tests/compare_to_comby/comby.sh 100000 test/context_tests/compare_to_comby/max_blocks/Comby/100000.txt 1000 grep maximum"
User time (seconds): 440.95
System time (seconds): 0.17
Percent of CPU this job got: 97%
Elapsed (wall clock) time (h:mm:ss or m:ss): 7:30.28
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 160660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 117
Minor (reclaiming a frame) page faults: 44452
Voluntary context switches: 94
Involuntary context switches: 2682
Swaps: 0
File system inputs: 0
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Loading